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ABSTRACT 

The purpose of this paper is to demonstrate in detail 
how the Empirical 3ayes (EB) statistical estimation strategy can be|^ 
applied to an important class of educational research contexts. EB 
.methods are tailored specif ically to the analysis of data with a 
hierarchical structure. For instance, investigators may be interested 
in discovering how effects within schools (e.g., the relationship 
between student social class and aclxievement ) vary as a function of 
differences between schools^ (e .g. , policies and practices). 
Similarly, meta-analysts often wii^h to find out how differences 
between experimental and control groups within studies vary as a 
function of differences between studies (e.g., how treatments are 
implemented). Developmental psychologists care about how children''s 
intellectual growth rates vary as a function of different pre-school 
experiences. In each case parameters at one level (within schools, 
within studies ,^ anci within children) vary as a function of parameters 
at another level (between schools, between studies, between 
children). This paper explains how the EB strategy works when the 
central goal of an investigation is to estimate the second level 
parameters (i.e., the between-group parameters), an4^ an important 
ancillary goal is to assess the adequacy of a hierarchical linear 
model for fitting such hierarchical data. (Autjior) 
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The purpose of this paper is to demonstrate rfn detail how the , 
''empirical Bayes" or "James-Stein** statistical estimation strategy / / 
(Lindley and Smith, 1972; Strenio* 1981; Strenio, Bryk, and Weisberg, 
1983; Raudenbush, 1984: note 1) can be applied to an important 
class of educational research contexts. Empirical Bayes (**EB" for short) 
methods are tailored specifically to the analysis of data with a 
hierarchcial structure. For instance, investl>gators may be interested 
in discovering how effects within schools (e.g., the relationship 
between student social class and achievement) vary as a fijnction of 
differences between schools (e.g., policies and practices). Similarly,' 
meta-analys ts^f ten wish to find out how "differences between experimental 
and control groups within studies vary as a function of differences between 
studies (e.g., how treatments are implemented). Developmental psychologists 
care about how children's intellectual growth rate§ va^ry as a function 
of different pre-Si^ool experiences.. In each case parameters at one level 
(within-schools , within studies, and within children) vary as a function 
of parameters at another level (between schools, between studies, between 
children). This ^aper explains how the EB strategy works 
when the central, goal of an investigation is to estimate the second 
level paramters. (i.e. , the between-group paramters) , and an important 
ancillary goal is to assess the adequacy of a hierarchcial linear model 
for fitting such hierarchical data. . , 



1. 




The purpose of this pflBM^ demonstrate In detalT^how the 



"empirical Bayes" or " Jaines^^[j|^^|^tatistlcal estimation stjrategy 
(Lindley and Smith, 1972; Strenl55|^^981; Strenio, Bryk, and Weisbergj 
1983; Raudenbush, 1984: note 1) can ^e applied to an important : • ^ ^ » . ^ 

class of educational research contexts. Empirical Bayes ("EB'v fp^' sh^ '"'y-^. 
methods are tailored^ specif icalj^^o the analy'sis of data with a- ^ 7/ 
hierarchcial structure. For instance, investigators may be interested 
in discovering how effects within schools , (ei. gv , the rel^tionshrip • ' ^ '-'''^i: 
between student, social class and achievement) vary. aS' aiifmctljbn ofy,^.-,:;'/:-^^^/:'.-^ 

differences between s'chools (e.g., policies^'^nd practices):; :Siiilila^lO^ 

^ ' j- ' ■ . 

meta-analysts often wish to find out how differences between- ^^experimental '.^ 

' ^V.' ■ V V:- ■-■ •• 

and control groups within studies vary as a f undtlon. of differences ^between 

■ ^ ' . ■ I 

Studies (e.g. , how treatments arp implemented)'.' Development^l-psycho'^logists' v 

' care about how children's intellectual growth rates vary> as ja .function - ; ^ 

■ * '■■ .\ J 

of different pre-school experiences. In each case parameters .^/^t - one leVel \ 

(within-Schools ,,,vithin studies, and within children) vary a§ a fimction 

of parameters at another level (between schools, between studies, be^tween 

x:hildren). This paper explains how the EE strategy works - '^ 

when the centVal goal of an investigation is to estimate the second 

level paramters (i.e.,- the between-group paramters) , and an Important 

•ancillary goal is to assess the adequacy of a hierarchcial linear model 

A / - 

for fitting such hierarchical data. 
' ■ *' . 

We proceed by explaining the empirical Bayes concept ' Goncretsely in the 
context of school effectiveness research. A reanalysis of the High School 
and Beyond data (Coleman, Hoffer, and Kilgore, 1982) demonstrates the 
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l^se rthe method v.in" simu between- lari^i wi'thifi-sctiool 



• lfe-tlib:feib^lca Research 



, : . ' A pen t rail Vid-if^i^^ school ef f ectiv^es^s . 

. .;:is: the;:myl;tiVI TKere are differences between 

Schools and ^differences amp'ng the sfcudents within j each ^slihool. , ^o^^ 
valid, statistical analyses mus accoimt simulfcanepusiy^ for effects fit both 
v'. levels.. The' 'most* plroraising current; approaches (Gooley , Bbn^,' arid >ko, 1981; 
. , Bu^ Millar, -1978);- first e;Stimate rejatdonslji^s/^^ student 

''Vbaclcgroiihd characteristics* and' student outcomes" "at each school; these 

• ^ ^ . _^ / -""A. ' ^ ^ 

. reUatiSnsJKips, as- by. regression coefficients, then serve, as , 

outcome va fi tables '^f or an assessment of the importance of school^pOlicfes 
::^nd practices* " ' . 

. ; H^^ major difficulties typically plague the ^ 

^'"slopes as oi:^omes" analysis:^ - 

■ ;1. Wi thin-school regression coefficients are typically estimated ^ - 

. unreliably: far mare unreliably than, sayj school means. Such sampling ' . , 
Variance is especially acut^ when sample sizes within schools ar*e siyiatl ; 
' and/or several regression coefficients per school are estimated. , ■ 

This unreliability of slopes as outcomes means that- even if some schools . * 
•are more effective than others in, say, minimizing the effect of student . 
social class oh achievement, the anialysis may fail to detect this . effect ■ 
' or seriously underestimate its importance. ^ . 

2. *'Slopes-as-out comes" research very often uses a single regression 
slope per school as an outcome variable. For instance, one might use 
the SES/achievement slope as an outcome. Yet it will typicallybe necessary to adjust 
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for other variables which ^ght confound the SES/achievement relationship. 



'l^weveri toLinijcludei'm schbois requires a statistica 

o^odel{\ljhp:J;y the interdiependejice. of these coefficients. 

.thev E& ap^pcdach helps the investigator overcome these difficulties 
. in^manyp iristiaiifel^s^^^^^^^^^ it enables tlie analyst' to diSJtiguish between 

ytwovsoui^^ of the within-school slopes: 

thV'varisDttbn o.fr'the pararAters themselv&s, which actually constitute the 

.■.-*(•..■• ■ ■ •- a. N ' /, » t 

V. ■ '"•••V' ■ <»■ ■ ■ .* . . ■ . ' v' , ■/• ' ■•• - • ' .. . . • . ■ ' 

Object-, of; the i^^ variance of the. estimates. 

Seqondi.-the E5 "a:ppraau:h^-'l^^ into ^count the unequal preMsion of the 
within-school ^aratn^ter; .estimates , optimally weighting them to minimize 

' variance .c^^i be tweeh--school estimates. Third, EB allows a fully multivariate 
. forrauJation' wh^ all estimates for the interdependenc^ among^ 

the 'mult'll>ie^,.^^ estimated within schools. Finally^, 

EB is very vflepdLble: sample sizes per school can vary greatly without 

"biasing estimates, without throwing out data or "plugging in" . guesses 

for missing values,- and the investigator can make a variety of assi^ptions 

of eiror variances, and coyariances. The key restriction is that data within 

schools are assumed normal; and within-school paramters ate assumed • 

to have a normal "prior" distri'Eutsion. Further the consequences of 

violating these assumptions are not yet well known. 

The strengths of^ the EB approach are illustrated by a reanalysis of 

the- HiRh School and Beyond data. 

The "Common S chool" Effect * " 
. r 

Recently Coleman eC al (1982) have inspired a re-examination of the 
"excellence vs. equity" issue in American High .Schools. They found that 

Catholic high schools promote higher achievement^ than public high schools 

(J 

and that this positive effect of Catholic schools was most pronounced; for 
lower SES students. We show how EB methods can be employed systematically 
to investigate this assertion. We begin with a simple two-stage model 
with SES as a predictor within schools and no predictors between schools. 



We then demonstrate how to a'dd predictors at each level until a model 
is discovered which adequately accounts for variation at both levels. 
Model I / V , 

Within school i (i=l,2, . .Mc=176) mathematics achievement varies as 

• 1 

a function of student social class and' random error: f 



Vit = mathematics achievement for student t in school ij. 

jtf = the mean math achievement for school i ;* - 
= the effect of SES on math achievement 
ui thin school i ; 
Xit = the SES of student t in school i; 
x\. = the mean. SES for school i; 

Rit = the error of estimate for student t in school i. 
Thus a large error indicates that knouledg^^of 
that student^s SES uas unhelpful in estimating 
the student's math score. 



^Between schools, we assume first that all -variation of within-schooJ^ 
parameters (i.e., all means and slopes) is randqm , variation around a grand 



mean; 



Hi = PL + Uoi 

Bi = B + Ui i . 



|i = the grand mean achievement across all schools; 
Uoi = the effect of school i an mean math achievement; 
S = the average effect of SES on achievement 

pooled uithin all schools; 
Uzi = the effect of school i on the SES/math rel ationshipj 
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By employing the EM algorithm (Dempster, Laird, and Rubin, 1977) 
it is possible. to estimate the proportion of variance of estimated 
school means and\ylopes which represents parameter variance as opposed to 
variance of errors of estimation. Clearly this, proportion has important 
imiplications for further study. For instance if virtually all variance 
of slope estimates were attributable to error, we would infer that 
schools are very, much alike in the strength of the SES/math relationship. 
It would then be of little use to hunt for school policies and 'practices 
which "explain" such variance. On the other hand if a substantial portion of 
variance is parameter variance, it makes good sense to search for such 
explanatory variables. Re-estijnation of these variance components then enables 
a reassessment of model adequacy. 

Model I'f ^ • , « ' 

Next we again estimate the same model as before "within schools," 
but now add a predictor "between schoolZ^^^^chool sector (0 = public, 
1 = Catholic). * • . . 
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The'seQond ^stage of the model now becomes 



Hi = Too + yoiM?. Uo? 



700 = the mean achievement for public schools; 

701 = the ^sactor" effect i'' the mean difference betueen 

) Catholic and publ ic schools on mean math achieve 

. * merit 

Uo ? = The discrepancy between school i^s mean and -^he 

mean for school i^s sector. 
Yio = the average effect of SES on math achievement 

within public shcools; 
Yii = The mean difference betueen public and Catholic 
schools on the strength of 
the SES effect within schools. 
Wi = sector: 0 if a school is public; 1 if Catholic. 
Ui 1 = the discrepancy between the effect of SES 

on achievement 

^ within school i and the average effect for 

school i's sector. 
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Model III , ■ 

>We now add a "wlthln-school" predictor: hours of homework per week. 
We add this factor because we hypothesize that it may attenuate the effect' 
of SES within schools, The^inodel between schools remains the same: with a single 
predictor, school sector. The wi thin-school model now becomes: 

(3.5) y i.t = »ii + Bi i(xi i t - xi 1.) + B2i(x2i t - Xzi .) + ill t 



Bii = effect of SES within school i; 
xi,-t = the SES of student t in school i; 
Bzi the effect hcmswork within school i 
XzM = hours per week of homework done by 
student t in school i. ^ 



r 



Mod^l IV 
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We now add two predictors between schools:^ the mein SES of the school 
and the mean SES by sector interaction effect; 

(3. 7a) • Ui = roo + roiMn + rozMzi + ro3Ui ;w2i + Uoi 

Bzi = 720 + 721 Wii + rzzWzi + rzoWiiUzi + Uzi 

. ' ' ' f . . - 

Here wzi is the mean SES of the students in school i, and yoz/ ri2> and*' 

722 are the effects of mean SES on mean math achievement, the SES/Math 

achievement slope, and the homework/math slope, respectively-. Alsq, 

roo* ri3> and rza* a're the effects of the mean SES-by-sector interac- 
tion; : 10 



We"^ stop here. Our purpose is to illustrate the us^ of i±e liiethbd \ 

■t- . ' U . - . 

for controlling variation w;itiiin and between schools, not to find the / 
optimal model. ' - ^ - i 

• ■ Results 

* ■ . ■ ■ 

The results of estimatings^the fouy models .are 'suimnarized in T^ble 1. 
Key findings are the following. ' . * 

Model I • . • . 

On average j^there is an unmistakable, linear relationship between 
student SES and mathematics achievement within schools, ^ result which 
is hardly surprising. There is, however, substantial variation among the 
schools in this effect, after removing that part of the variation 

among the estimated slopes solely attributable to their unreliability of 
estimation. In fact, about 35% of the total variance d'f the estimated 
slopes is estimated to reflect variation among the parameters. 

The relative unreliability of the slopes as outcomes is illustrated, 
however, by the fact that, in contrast, 92% of the variance <if the' school 
means is estimated to be systematic, that is, to be variance of the parameters. 
Model II ^ ■ 

Catholic schools are found to have 1) ^substantially higher mean achieyement 
than public*schools ; and 2) substantially smaller slopes , illustrating 
the egalitarian effect found by Coleman and his associates. These results 
are illustrated graphically in Figure 1. - * ' ( 

Inclusion of sector accounts for 71.6% of the original estimated 
variation of the slope parameters, but because of the' unreliability of J 
these estimates, only 25% of the totaL variance. Inclusion of sector is 
less helpful- in explaining variation in school means: 11.3% of the parameter 
variance and 10.2% of the total variance is explained. 



Model III v~ 

;. ' ^ Y , ■ ^ t ■! . V • 

V Inclusion or 'homework within school^ 1) helps very modestly in 
explaining ^ithin-school variation; and 2) leads^to a very small 
adjustment of the efjipct of sector on slopes, The^ egalitarian' effect 
of Catholic fiphools remains largely intact^. ^ ' 

■ Model IV - \. • . ^ , 

^. inclusions of t'he meax^^SES and ^the mean S?S-by-sectc/r interaction ^ 
has /several important effects on estimates: ' ^ 

1. The. CathoXic school advantage in mean ^ath achievement disappears. 

2\ The "egalitarian effect" of Catholic schools remains intact : . 
SES e%e cts witTilti Catholic schools remain substantially -^^maller, on' average, 
than those within public schpols. ) 

3. . Combijjing evidence from 1) and 2) yields file inference of a 

disordinal interaction between school sector and p^upils' SES; * , 

Catholic 'schools appear to benefit poorer students but 'td^ ijenalize more 

■ . • ' ... ' ' . ^ ■ ■ ' ^ 

advantaged ..students. This, i^nference^ like^others , is quite tent;^ative 

sindB we! might plausibly revise these estimates in light of new information 

yielded by more complex models. 

4. A substantial proportion of ^ the variance, in the slope parameters has 

■w 

been explained: 83.2%. (Only 29.0% of the. total slope variance" — which include 
^'error variance --has-been e^tplained. > Th^. comparable figures for school' 

if ' 

means are l^s substantial but still impressive: 66.2% and 60.0%. 

Lt is now ffossible, in fact, to^ retain' the null hypothesis that 
>air variance among ^he slope, paramters has been explained. We employ the fit 
st^istic proposed by Hedges (1982): 



jU/ViCBn - 710* - ri1*Mli -yi2*U2i - yi3*HiiU2i)2 



11. . • • 

where denotes the- sampling variance of the SES/math slope \for 
school i. If we assume that > these estimated variances are equivalent to 
their true values, this statistic has a chi-squared distribution with 
:k-4 = 172 degrees of fire^do^mr; !^ case the statistic has a value 

of 193.07, which is eqxiivaient^t normal deviate of 1.13. / 

■ ■ • ' ' ' 't ■ _ • 

Conclusions . 

This paper has illustrate^d ^ systematic approach for adjusting simultan'e- ' 

ously for effects measured ar two levels of aggregation. It has also • - . 

shown how to assess model adequacy, in each stage of an iterative process 

r 

whereby variables "within" and "between" are added. This strategy, known 
as empirical Bayes estimation, has obvious attractions for educational 
researchers who cfbmmonly confront multi-level data which has proved 
resistant to satisfactory quantitative assessment. We showed , fo^ instance, 
how to resolve the regression of math achievement on SES into three components 
a between-school component; a pooled within-school component; and a school- 
specific component. Further, the estimates of each of these components, 
was adjusted simultaneously for potentially confounding variables within 
and between schools. This simultaneous, multi-level strategy offers 
many opportunities for gains in educational research. 

Analysts are advised, however, to consider the tenability of key 
assumptions: that within-school data are normally distributed, and that 
the parameters whose variance is to be explained are normally distributed. 
Under these conditions the estimates used here are maximum likelihood 
estimates with concomitant advantages of asymptotic efficiency and known 
asymptotic distributions. Little is yet known about the consequences of 
violating these assumptions. 

• * 
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Table 1. 



Suaaary of Reaulta 



FIXED EFFECTS 



Model 1 
(baaic) 

affect 



a.e. 



Model 2 
(SECTOR e 
Btage 2) 
affect a.e. 



Model 3 
(HMWORK e 
atage 1) 
effect a.e.* 



Model 4 
(SchSES ♦ 
int. e atage 1) 
effect s.e. 



Overall Effects ^ 
grand .aean MATHACT* X,, ^ 
■ean SES/KATHACH relation. »*S 
■ean HMWORK/MATH^H rel.^-Jf^^'* 

Sector Effecta 
■aafi'KATHACH,^^ 
SES/MATHACH relation., I'J, 
HMWORK/MATHACH relation., 

School SES Effects 
■ean MATHAGH, 
SES/MATHACH relation . , H 
HMWORK/MATHACH relation.., "l*" 

School SES X SECTOR Effecta 
■ean MATHACH, "4^* 
SES/MATHACH relation. 
HMWORK/MATHACH relation., V* 



100.74 
4.52 



.573 
.290" 



5.06 

-3.86 



1.09 
.49 



5.06 

-3. 56 
-.33 



1.09 
.49 
.10 



-4:24 
-.36 



15.09 
2.32 
.11 



-2.50 
.41 
-.39 



.63 
.45 
.11 



1.32 
.90 
.19 



1.85 
1.28 
.25 



RANDOM EFFECTS 

variation axong school aeans 
in MATHACH 
eat. var ( ) 
eat. var ( I > 

X of variance ayateaatic 



.1 



va 

Si 



^ion aaong SES/MATHACH 
lopea across schoola 
eat. var ( 
eat. var <5,;\'*»t^ 

X of variance ayateaatic 



53.25 
58.66 

0.92 



5.26 
15.11 

0.35 



^ R"^ 



t. _ 



R*- 
R** 



47.36 

0.102 
0.113 



1.49 

0.250 
0.716 



R"^ 
R* 



47.24 

0.102 
0.113 



1.48 

» 0.252 
' 0.719 



18.01 

0.600 
0.662 



0.88 

0.290 
0.832 



Note, nuaerous oth 
could be eatiaated 




variances and covariancea aaong the paraaetera 
pending upon the aubstantive problea of interest. 
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Figure 1. 
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